Field variable tone mapping for 360 content

ABSTRACT

Field variable tone mapping is performed for 360 content. An image capture device includes an image sensor and a processor. The image sensor obtains a hyper-hemispherical image and the processor performs local tone mapping (LTM) on a first area of the hyper-hemispherical image and performs global tone mapping (GTM) on a second area of the hyper-hemispherical image to obtain a processed image. The processor may be configured to display, store, output, or transmit the processed image.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.17/749,689, filed May 20, 2022, which is a continuation of U.S.application Ser. No. 16/843,301, filed on Apr. 8, 2020, now U.S. Pat.No. 11,350,042, the entire disclosures of which are incorporated byreference herein.

TECHNICAL FIELD

This disclosure relates to image processing. In particular, thisdisclosure relates to tone mapping for multi-image sensor content.

BACKGROUND

One of the most efficient classes of tone mapping algorithms is localtone mapping (LTM). LTM, however, requires a wide neighborhood to becomputed, and is therefore not suitable for 360 degree (360) content. Onpairs of fish-eye or hyper-hemispherical images, the quality and thecontinuity of the processing across the stitch line cannot be guaranteedusing LTM since the pixels outside the image circle may modify theexpected output. In equi-rectangular projection (ERP) images, thedistortion on the poles may be too large, and would require afield-variable kernel that would be theoretically infinitely wide on thepoles. In equi-angular cubemap (EAC) images, the continuity on the edgesof both bands may not be guaranteed using LTM. Accordingly, devices andmethods for performing LTM on 360 content is desired.

SUMMARY

Disclosed herein are implementations of image capture devices andmethods for field variable tone mapping of 360 content. In an aspect, animage capture device may comprise an image sensor. The image sensor maybe a hyper-hemispherical image sensor. The image sensor may beconfigured to capture a hyper-hemispherical image. Thehyper-hemispherical image may include an image circle portion. Acircumference of the image circle portion may represent a stitch linebetween the hyper-hemispherical image and a second hyper-hemisphericalimage. The second hyper-hemispherical image may be obtained from asecond image sensor. The image capture device may include a processor.The processor may be configured to perform LTM on a first area of theimage circle portion. The processor may be configured to perform globaltone mapping (GTM) on a second area of pixels of the image circleportion. The GTM may be performed on a portion of a predefined area ofpixels overlaps with the stitch line. The processor may be configured tostitch the hyper-hemispherical image and the second hyper-hemisphericalimage at the stitch line to obtain a processed image. The processor maybe configured to display, store, output, or transmit the processedimage.

In another aspect, a method may include obtaining a hyper-hemisphericalimage that includes an image circle portion. A circumference of theimage circle portion may represent a stitch line between thehyper-hemispherical image and a second hyper-hemispherical image. Themethod may include dividing the hyper-hemispherical image into aplurality of blocks. The method may include determining whether a blockof the plurality of blocks contains a portion of the image circleportion. On a condition that the block contains a portion of the imagecircle portion, the method may determine whether the block overlaps withthe stitch line. On a condition that the block does not overlap with thestitch line, the method may compute a neighborhood luminance of a pixel.The method may include applying a gain to the pixel based on theneighborhood luminance. On a condition that the block overlaps with thestitch line, the method may include determining a distance of the pixelfrom a center of the image circle portion. The method may includecomputing a luminance of the pixel. The method may include applying again to the pixel based on the luminance. The method may includeapplying an attenuation map to the pixel based on the determineddistance of the pixel from the center of the image circle portion. Themethod may include stitching the hyper-hemispherical image and thesecond hyper-hemispherical image at their respective stitch lines toobtain a processed image. The method may include displaying, storing,outputting, or transmitting the processed image.

In another aspect, an image capture device may include a first imagesensor, a second image sensor, and a processor. The first image sensormay be configured to capture a first hyper-hemispherical image. Thefirst hyper-hemispherical image may include a first image circleportion. A circumference of the first image circle portion may representa first stitch line. The second image sensor may be configured tocapture a second hyper-hemispherical image. The secondhyper-hemispherical image may include a second image circle portion. Acircumference of the second image circle portion may represent a secondstitch line. The processor may be configured to perform LTM on a firstarea of pixels of the first image circle portion. The processor may beconfigured to perform GTM on a second area of pixels of the first imagecircle portion, wherein the second area of pixels is a predefined areaof pixels that overlaps with the stitch line. The processor may beconfigured to stitch the first hyper-hemispherical image and the secondhyper-hemispherical image at the first and second stitch lines to obtaina processed image. The processor may be configured to display, store,output, or transmit the processed image.

In another aspect, an image capture device may comprise an image sensor.The image sensor may be a hyper-hemispherical image sensor. The imagesensor may be configured to capture a hyper-hemispherical image. Thehyper-hemispherical image may include an image circle portion and a darkcorner portion. A circumference of the image circle portion mayrepresent a stitch line between the hyper-hemispherical image and asecond hyper-hemispherical image. The second hyper-hemispherical imagemay be obtained from a second image sensor. The image capture device mayinclude a processor. The processor may be configured to perform LTM on afirst area of the image circle portion. The first area of pixels mayhave a first radius relative to a center of the image circle portion.The processor may be configured to perform global tone mapping (GTM) ona second area of pixels of the image circle portion. The GTM may beperformed on a condition that a portion of a predefined area of pixelsoverlaps with the stitch line. The second area of pixels may have asecond radius relative to the center of the image circle. The secondradius may be greater than the first radius. The processor may beconfigured to stitch the hyper-hemispherical image and the secondhyper-hemispherical image at the stitch line to obtain a processedimage. The processor may be configured to display, store, output, ortransmit the processed image.

In another aspect, a method may include obtaining a hyper-hemisphericalimage that includes an image circle portion and a dark corner portion. Acircumference of the image circle portion may represent a stitch linebetween the hyper-hemispherical image and a second hyper-hemisphericalimage. The method may include dividing the hyper-hemispherical imageinto a plurality of blocks. Each block of the plurality of blockscontains a predetermined number of pixels. The method may includedetermining whether a block of the plurality of blocks contains aportion of the image circle portion. On a condition that the blockcontains a portion of the image circle portion, the method may determinewhether the block overlaps with the stitch line. On a condition that theblock does not overlap with the stitch line, the method may compute aneighborhood luminance of a pixel of the predetermined number of pixels.The method may include applying a gain to the pixel based on theneighborhood luminance. On a condition that the block overlaps with thestitch line, the method may include determining a distance of the pixelfrom a center of the image circle portion. The method may includecomputing a luminance of the pixel. The method may include applying again to the pixel based on the luminance. The method may includeapplying an attenuation map to the pixel based on the determineddistance of the pixel from the center of the image circle portion. Themethod may include stitching the hyper-hemispherical image and thesecond hyper-hemispherical image at their respective stitch lines toobtain a processed image. The method may include displaying, storing,outputting, or transmitting the processed image.

In another aspect, an image capture device may include a first imagesensor, a second image sensor, and a processor. The first image sensormay be configured to capture a first hyper-hemispherical image. Thefirst hyper-hemispherical image may include a first image circle portionand a first dark corner portion. A circumference of the first imagecircle portion may represent a first stitch line. The second imagesensor may be configured to capture a second hyper-hemispherical image.The second hyper-hemispherical image may include a second image circleportion and a second dark corner portion. A circumference of the secondimage circle portion may represent a second stitch line. The processormay be configured to perform LTM on a first area of pixels of the firstimage circle portion. The first area of pixels may have a first radiusrelative to a center of the first image circle portion. The processormay be configured to perform GTM on a second area of pixels of the firstimage circle portion on a condition that a portion of a predefined areaof pixels overlaps with the stitch line. The second area of pixels mayhave a second radius relative to the center of the first image circleportion. The second radius may be greater than the first radius. Theprocessor may be configured to stitch the first hyper-hemisphericalimage and the second hyper-hemispherical image at the first and secondstitch lines to obtain a processed image. The processor may beconfigured to display, store, output, or transmit the processed image.

In another aspect, an image capture device may include an image sensorand a processor. The image sensor may be configured to obtain ahyper-hemispherical image. The processor may be configured to performLTM on a first area of pixels of the hyper-hemispherical image. Theprocessor may be configured to perform GTM on a second area of pixels ofthe hyper-hemispherical image to obtain a processed image. The amount ofLTM performed may progressively converge to an amount of GTM performedin a third area of pixels that is between the first area of pixels andthe second area of pixels. The processor may be configured to output theprocessed image.

In another aspect, a method may include obtaining a hyper-hemisphericalimage that has a first image portion. The method may include dividingthe hyper-hemispherical image into a plurality of blocks. The method mayinclude determining whether a block of the plurality of blocks containsa portion of the first image portion. The method may include determiningwhether the block overlaps a second image portion. The method mayinclude computing a neighborhood luminance of a pixel and applying again to the pixel based on the neighborhood luminance when the blockdoes not overlap with the second image portion. The method may include,when the block overlaps with the second image portion, determining adistance of the pixel from a center of the first image portion. Themethod may include, when the block overlaps with the second imageportion, computing a luminance of the pixel. The method may include,when the block overlaps with the second image portion, applying a gainto the pixel based on the luminance. The method may include, when theblock overlaps with the second image portion, applying an attenuationmap to the pixel based on the determined distance of the pixel from thecenter of the first image portion to obtain a processed image. Themethod may include, when the block overlaps with the second imageportion, outputting the processed image.

In another aspect, an image capture device may include a first imagesensor, a second image sensor, and a processor. The first image sensormay be configured to obtain a first hyper-hemispherical image that has afirst image portion. The second image sensor may be configured to obtaina second hyper-hemispherical image that has a second image portion. Theprocessor may be configured to perform LTM on a first area of pixels ofthe first image portion. The processor may be configured to perform GTMon a second area of pixels of the first image portion to obtain aprocessed image. The amount of LTM performed may converge to an amountof GTM performed. The processor may be configured to output theprocessed image.

In one or more aspects, the processor may be configured to perform LTMon each pixel of the first area of pixels based on the predefined areaof pixels. In one or more aspects, the predefined area of pixels may beapproximately a 100×100 pixel area. In one or more aspects, theprocessor may be configured to perform LTM on each pixel of the firstarea of pixels to remove low frequency variations. In one or moreaspects, the processor may be configured to perform LTM on each pixel ofthe first area of pixels to preserve or enhance high frequency details.In one or more aspects, the processor may be configured to output theprocessed image in an equi-angular cubemap (EAC) format. In one or moreaspects, the processor may be configured to output the processed imagein a stitched pair fish-eye format. In one or more aspects, theprocessor may be configured to divide the hyper-hemispherical imagesinto a plurality of blocks that contain a predetermined number ofpixels. In one or more aspects, the neighborhood luminance may be anaverage luminance of the predetermined number of pixels. In one or moreaspects, the average luminance may be a weighted average of theluminance of the predetermined number of pixels. In one or more aspects,each block may have approximately a 100×100 pixel area. In one or moreaspects, the attenuation map may be applied to a smoothed luminance ofthe pixel. In one or more aspects, the attenuation map may be applied toa local contrast enhancement strength of the pixel.

In one or more aspects, LTM may be performed on a third area of pixelsof the second image circle portion. The third area of pixels may have athird radius relative to a center of the second image circle portion. Inone or more aspects, GTM may be performed on a fourth area of pixels ofthe second image circle portion on a condition that a portion of thepredefined area of pixels overlaps with the second stitch line. Thefourth area of pixels may have a fourth radius relative to the center ofthe second image circle portion. The fourth radius may be greater thanthe third radius. In one or more aspects, the processor may beconfigured to perform LTM on each pixel of the first area of pixels andthe third area of pixels based on the predefined area of pixels. In oneor more aspects, the processor may be configured to perform LTM on eachpixel of the first area of pixels and the third area of pixels to removelow frequency variations. In one or more aspects, the processor may beconfigured to perform LTM on each pixel of the first area of pixels andthe third area of pixels to preserve or enhance high frequency details.In one or more aspects, the processor may be configured to perform acombination of LTM and GTM on a fifth area of pixels of the first imagecircle portion. The fifth area of pixels may be between the first areaof pixels and the second area of pixels. In one or more aspects, thecombination may be based on the attenuation map.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIGS. 1A-1B are isometric views of an example of an image capturedevice.

FIGS. 2A-2B are isometric views of another example of an image capturedevice.

FIG. 2C is a top view of the image capture device of FIGS. 2A-2B.

FIG. 2D is a partial cross-sectional view of the image capture device ofFIG. 2C.

FIG. 3 is a block diagram of electronic components of an image capturedevice.

FIG. 4 is a diagram of an example of a tone mapping process applied in apair of fish-eye images.

FIG. 5 is a graph of an example of a radial profile of an attenuationmap.

FIG. 6 is a diagram of an example of a tone mapping process applied inan equi-angular cubemap (EAC) format image.

FIG. 7 is a flow diagram of an example of a method for field variabletone mapping.

FIG. 8 is a flow diagram of an example of another method for fieldvariable tone mapping.

DETAILED DESCRIPTION

In a dual-lens image capture device example, RAW fish-eye imagesrespective two circular images surrounded by dark pixels, one circularimage for the front lens and the second circular image for the backlens. Valid pixels are contained into an area that may be referred to asan image circle. Applying GTM on these images is possible and currentlyperformed in image capture devices. GTM with identical parameters onboth images does not prevent from correctly stitching the front and backimages for the generation of 360 content stored in specific formats,such as EAC and equi-rectangular projection (ERP). An EAC image formatincludes two bands of a cube where each band is composed of three cubefaces. The 360 content from two hyper-hemispherical image sensors may beprojected on this cube. The external edges of each band represent therespective stitch lines for each band. The two bands may be stored asrectangular images and stored in a single EAC image format by verticalconcatenation.

LTM application is more delicate on 360 content than single rectangularimages because of a continuity constraint between the two fish-eyeimages, which may not be guaranteed by the local filtering appliedindependently on each image. LTM may create artifacts close to the outeredge of the image circle because of the surrounding dark pixels. Onrectangular images, an image extension may be performed, however this isnot possible on circular images, and would be computationally expensive.In addition, stitching the content requires similar content, for examplesimilar luminance, similar chrominance, or both, near the stitch line.Applying LTM to each image separately would lead to undesirable imagequality near the stitch line.

The implementations described herein enable local tone mappingprocessing on 360 content while processing any 360 image format such as,for example, fish-eye pairs or EAC format images, by applying LTMprocessing in the areas that are not impacted by border or continuityissues between bands or lenses, for example, in the center of eachfish-eye image. The implementations described herein may progressivelyconverge toward GTM with shared parameters between the two hemispheresof a fish-eye pair or the two bands of an EAC image when approaching thearea where continuity is required for stitching. Accordingly, theresulting 360 image will show no differences in the areas near thestitch line where it is necessary to have the same kind of processing,and the benefit of LTM is preserved in other areas of the image.

FIGS. 1A-1B are isometric views of an example of an image capture device100. The image capture device 100 may include a body 102, a lens 104structured on a front surface of the body 102, various indicators on thefront surface of the body 102 (such as light-emitting diodes (LEDs),displays, and the like), various input mechanisms (such as buttons,switches, and/or touch-screens), and electronics (such as imagingelectronics, power electronics, etc.) internal to the body 102 forcapturing images via the lens 104 and/or performing other functions. Thelens 104 is configured to receive light incident upon the lens 104 andto direct received light onto an image sensor internal to the body 102.The image capture device 100 may be configured to capture images andvideo and to store captured images and video for subsequent display orplayback.

The image capture device 100 may include an LED or another form ofindicator 106 to indicate a status of the image capture device 100 and aliquid-crystal display (LCD) or other form of a display 108 to showstatus information such as battery life, camera mode, elapsed time, andthe like. The image capture device 100 may also include a mode button110 and a shutter button 112 that are configured to allow a user of theimage capture device 100 to interact with the image capture device 100.For example, the mode button 110 and the shutter button 112 may be usedto turn the image capture device 100 on and off, scroll through modesand settings, and select modes and change settings. The image capturedevice 100 may include additional buttons or interfaces (not shown) tosupport and/or control additional functionality.

The image capture device 100 may include a door 114 coupled to the body102, for example, using a hinge mechanism 116. The door 114 may besecured to the body 102 using a latch mechanism 118 that releasablyengages the body 102 at a position generally opposite the hingemechanism 116. The door 114 may also include a seal 120 and a batteryinterface 122. When the door 114 is an open position, access is providedto an input-output (I/O) interface 124 for connecting to orcommunicating with external devices as described below and to a batteryreceptacle 126 for placement and replacement of a battery (not shown).The battery receptacle 126 includes operative connections (not shown)for power transfer between the battery and the image capture device 100.When the door 114 is in a closed position, the seal 120 engages a flange(not shown) or other interface to provide an environmental seal, and thebattery interface 122 engages the battery to secure the battery in thebattery receptacle 126. The door 114 can also have a removed position(not shown) where the entire door 114 is separated from the imagecapture device 100, that is, where both the hinge mechanism 116 and thelatch mechanism 118 are decoupled from the body 102 to allow the door114 to be removed from the image capture device 100.

The image capture device 100 may include a microphone 128 on a frontsurface and another microphone 130 on a side surface. The image capturedevice 100 may include other microphones on other surfaces (not shown).The microphones 128, 130 may be configured to receive and record audiosignals in conjunction with recording video or separate from recordingof video. The image capture device 100 may include a speaker 132 on abottom surface of the image capture device 100. The image capture device100 may include other speakers on other surfaces (not shown). Thespeaker 132 may be configured to play back recorded audio or emit soundsassociated with notifications.

A front surface of the image capture device 100 may include a drainagechannel 134. A bottom surface of the image capture device 100 mayinclude an interconnect mechanism 136 for connecting the image capturedevice 100 to a handle grip or other securing device. In the exampleshown in FIG. 1B, the interconnect mechanism 136 includes foldingprotrusions configured to move between a nested or collapsed position asshown and an extended or open position (not shown) that facilitatescoupling of the protrusions to mating protrusions of other devices suchas handle grips, mounts, clips, or like devices.

The image capture device 100 may include an interactive display 138 thatallows for interaction with the image capture device 100 whilesimultaneously displaying information on a surface of the image capturedevice 100.

The image capture device 100 of FIGS. 1A-1B includes an exterior thatencompasses and protects internal electronics. In the present example,the exterior includes six surfaces (i.e. a front face, a left face, aright face, a back face, a top face, and a bottom face) that form arectangular cuboid. Furthermore, both the front and rear surfaces of theimage capture device 100 are rectangular. In other embodiments, theexterior may have a different shape. The image capture device 100 may bemade of a rigid material such as plastic, aluminum, steel, orfiberglass. The image capture device 100 may include features other thanthose described here. For example, the image capture device 100 mayinclude additional buttons or different interface features, such asinterchangeable lenses, cold shoes, and hot shoes that can addfunctional features to the image capture device 100.

The image capture device 100 may include various types of image sensors,such as charge-coupled device (CCD) sensors, active pixel sensors (APS),complementary metal-oxide-semiconductor (CMOS) sensors, N-typemetal-oxide-semiconductor (NMOS) sensors, and/or any other image sensoror combination of image sensors.

Although not illustrated, in various embodiments, the image capturedevice 100 may include other additional electrical components (e.g., animage processor, camera system-on-chip (SoC), etc.), which may beincluded on one or more circuit boards within the body 102 of the imagecapture device 100.

The image capture device 100 may interface with or communicate with anexternal device, such as an external user interface device (not shown),via a wired or wireless computing communication link (e.g., the I/Ointerface 124). Any number of computing communication links may be used.The computing communication link may be a direct computing communicationlink or an indirect computing communication link, such as a linkincluding another device or a network, such as the internet, may beused.

In some implementations, the computing communication link may be a Wi-Filink, an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBeelink, a near field communications (NFC) link, such as an ISO/IEC 20643protocol link, an Advanced Network Technology interoperability (ANT+)link, and/or any other wireless communications link or combination oflinks.

In some implementations, the computing communication link may be an HDMIlink, a USB link, a digital video interface link, a display portinterface link, such as a Video Electronics Standards Association (VESA)digital display interface link, an Ethernet link, a Thunderbolt link,and/or other wired computing communication link.

The image capture device 100 may transmit images, such as panoramicimages, or portions thereof, to the external user interface device viathe computing communication link, and the external user interface devicemay store, process, display, or a combination thereof the panoramicimages.

The external user interface device may be a computing device, such as asmartphone, a tablet computer, a phablet, a smart watch, a portablecomputer, personal computing device, and/or another device orcombination of devices configured to receive user input, communicateinformation with the image capture device 100 via the computingcommunication link, or receive user input and communicate informationwith the image capture device 100 via the computing communication link.

The external user interface device may display, or otherwise present,content, such as images or video, acquired by the image capture device100. For example, a display of the external user interface device may bea viewport into the three-dimensional space represented by the panoramicimages or video captured or created by the image capture device 100.

The external user interface device may communicate information, such asmetadata, to the image capture device 100. For example, the externaluser interface device may send orientation information of the externaluser interface device with respect to a defined coordinate system to theimage capture device 100, such that the image capture device 100 maydetermine an orientation of the external user interface device relativeto the image capture device 100.

Based on the determined orientation, the image capture device 100 mayidentify a portion of the panoramic images or video captured by theimage capture device 100 for the image capture device 100 to send to theexternal user interface device for presentation as the viewport. In someimplementations, based on the determined orientation, the image capturedevice 100 may determine the location of the external user interfacedevice and/or the dimensions for viewing of a portion of the panoramicimages or video.

The external user interface device may implement or execute one or moreapplications to manage or control the image capture device 100. Forexample, the external user interface device may include an applicationfor controlling camera configuration, video acquisition, video display,or any other configurable or controllable aspect of the image capturedevice 100.

The user interface device, such as via an application, may generate andshare, such as via a cloud-based or social media service, one or moreimages, or short video clips, such as in response to user input. In someimplementations, the external user interface device, such as via anapplication, may remotely control the image capture device 100 such asin response to user input.

The external user interface device, such as via an application, maydisplay unprocessed or minimally processed images or video captured bythe image capture device 100 contemporaneously with capturing the imagesor video by the image capture device 100, such as for shot framing orlive preview, and which may be performed in response to user input. Insome implementations, the external user interface device, such as via anapplication, may mark one or more key moments contemporaneously withcapturing the images or video by the image capture device 100, such aswith a tag or highlight in response to a user input or user gesture.

The external user interface device, such as via an application, maydisplay or otherwise present marks or tags associated with images orvideo, such as in response to user input. For example, marks may bepresented in a camera roll application for location review and/orplayback of video highlights.

The external user interface device, such as via an application, maywirelessly control camera software, hardware, or both. For example, theexternal user interface device may include a web-based graphicalinterface accessible by a user for selecting a live or previouslyrecorded video stream from the image capture device 100 for display onthe external user interface device.

The external user interface device may receive information indicating auser setting, such as an image resolution setting (e.g., 3840 pixels by2160 pixels), a frame rate setting (e.g., 60 frames per second (fps)), alocation setting, and/or a context setting, which may indicate anactivity, such as mountain biking, in response to user input, and maycommunicate the settings, or related information, to the image capturedevice 100.

The image capture device 100 may be used to implement some or all of thetechniques described in this disclosure, such as the technique 700described in FIG. 7 or technique 800 described in FIG. 8 .

FIGS. 2A-2B illustrate another example of an image capture device 200.The image capture device 200 includes a body 202 and two camera lenses204 and 206 disposed on opposing surfaces of the body 202, for example,in a back-to-back configuration, Janus configuration, or offset Janusconfiguration. The body 202 of the image capture device 200 may be madeof a rigid material such as plastic, aluminum, steel, or fiberglass.

The image capture device 200 includes various indicators on the front ofthe surface of the body 202 (such as LEDs, displays, and the like),various input mechanisms (such as buttons, switches, and touch-screenmechanisms), and electronics (e.g., imaging electronics, powerelectronics, etc.) internal to the body 202 that are configured tosupport image capture via the two camera lenses 204 and 206 and/orperform other imaging functions.

The image capture device 200 includes various indicators, for example,LEDs 208, 210 to indicate a status of the image capture device 100. Theimage capture device 200 may include a mode button 212 and a shutterbutton 214 configured to allow a user of the image capture device 200 tointeract with the image capture device 200, to turn the image capturedevice 200 on, and to otherwise configure the operating mode of theimage capture device 200. It should be appreciated, however, that, inalternate embodiments, the image capture device 200 may includeadditional buttons or inputs to support and/or control additionalfunctionality.

The image capture device 200 may include an interconnect mechanism 216for connecting the image capture device 200 to a handle grip or othersecuring device. In the example shown in FIGS. 2A and 2B, theinterconnect mechanism 216 includes folding protrusions configured tomove between a nested or collapsed position (not shown) and an extendedor open position as shown that facilitates coupling of the protrusionsto mating protrusions of other devices such as handle grips, mounts,clips, or like devices.

The image capture device 200 may include audio components 218, 220, 222such as microphones configured to receive and record audio signals(e.g., voice or other audio commands) in conjunction with recordingvideo. The audio component 218, 220, 222 can also be configured to playback audio signals or provide notifications or alerts, for example,using speakers. Placement of the audio components 218, 220, 222 may beon one or more of several surfaces of the image capture device 200. Inthe example of FIGS. 2A and 2B, the image capture device 200 includesthree audio components 218, 220, 222, with the audio component 218 on afront surface, the audio component 220 on a side surface, and the audiocomponent 222 on a back surface of the image capture device 200. Othernumbers and configurations for the audio components are also possible.

The image capture device 200 may include an interactive display 224 thatallows for interaction with the image capture device 200 whilesimultaneously displaying information on a surface of the image capturedevice 200. The interactive display 224 may include an I/O interface,receive touch inputs, display image information during video capture,and/or provide status information to a user. The status informationprovided by the interactive display 224 may include battery power level,memory card capacity, time elapsed for a recorded video, etc.

The image capture device 200 may include a release mechanism 225 thatreceives a user input to in order to change a position of a door (notshown) of the image capture device 200. The release mechanism 225 may beused to open the door (not shown) in order to access a battery, abattery receptacle, an I/O interface, a memory card interface, etc. (notshown) that are similar to components described in respect to the imagecapture device 100 of FIGS. 1A and 1B.

In some embodiments, the image capture device 200 described hereinincludes features other than those described. For example, instead ofthe I/O interface and the interactive display 224, the image capturedevice 200 may include additional interfaces or different interfacefeatures. For example, the image capture device 200 may includeadditional buttons or different interface features, such asinterchangeable lenses, cold shoes, and hot shoes that can addfunctional features to the image capture device 200.

FIG. 2C is a top view of the image capture device 200 of FIGS. 2A-2B andFIG. 2D is a partial cross-sectional view of the image capture device200 of FIG. 2C. The image capture device 200 is configured to capturespherical images, and accordingly, includes a first image capture device226 and a second image capture device 228. The first image capturedevice 226 defines a first field-of-view 230 and includes the lens 204that receives and directs light onto a first image sensor 232.Similarly, the second image capture device 228 defines a secondfield-of-view 234 and includes the lens 206 that receives and directslight onto a second image sensor 236. To facilitate the capture ofspherical images, the image capture devices 226 and 228 (and relatedcomponents) may be arranged in a back-to-back (Janus) configuration suchthat the lenses 204, 206 face in generally opposite directions.

The fields-of-view 230, 234 of the lenses 204, 206 are shown above andbelow boundaries 238, 240 indicated in dotted line. Behind the firstlens 204, the first image sensor 232 may capture a firsthyper-hemispherical image plane from light entering the first lens 204,and behind the second lens 206, the second image sensor 236 may capturea second hyper-hemispherical image plane from light entering the secondlens 206.

One or more areas, such as blind spots 242, 244 may be outside of thefields-of-view 230, 234 of the lenses 204, 206 so as to define a “deadzone.” In the dead zone, light may be obscured from the lenses 204, 206and the corresponding image sensors 232, 236, and content in the blindspots 242, 244 may be omitted from capture. In some implementations, theimage capture devices 226, 228 may be configured to minimize the blindspots 242, 244.

The fields-of-view 230, 234 may overlap. Stitch points 246, 248 proximalto the image capture device 200, that is, locations at which thefields-of-view 230, 234 overlap, may be referred to herein as overlappoints or stitch points. Content captured by the respective lenses 204,206 that is distal to the stitch points 246, 248 may overlap.

Images contemporaneously captured by the respective image sensors 232,236 may be combined to form a combined image. Generating a combinedimage may include correlating the overlapping regions captured by therespective image sensors 232, 236, aligning the captured fields-of-view230, 234, and stitching the images together to form a cohesive combinedimage.

A slight change in the alignment, such as position and/or tilt, of thelenses 204, 206, the image sensors 232, 236, or both, may change therelative positions of their respective fields-of-view 230, 234 and thelocations of the stitch points 246, 248. A change in alignment mayaffect the size of the blind spots 242, 244, which may include changingthe size of the blind spots 242, 244 unequally.

Incomplete or inaccurate information indicating the alignment of theimage capture devices 226, 228, such as the locations of the stitchpoints 246, 248, may decrease the accuracy, efficiency, or both ofgenerating a combined image. In some implementations, the image capturedevice 200 may maintain information indicating the location andorientation of the lenses 204, 206 and the image sensors 232, 236 suchthat the fields-of-view 230, 234, the stitch points 246, 248, or bothmay be accurately determined; the maintained information may improve theaccuracy, efficiency, or both of generating a combined image.

The lenses 204, 206 may be laterally offset from each other, may beoff-center from a central axis of the image capture device 200, or maybe laterally offset and off-center from the central axis. As compared toimage capture devices with back-to-back lenses, such as lenses alignedalong the same axis, image capture devices including laterally offsetlenses may include substantially reduced thickness relative to thelengths of the lens barrels securing the lenses. For example, theoverall thickness of the image capture device 200 may be close to thelength of a single lens barrel as opposed to twice the length of asingle lens barrel as in a back-to-back lens configuration. Reducing thelateral distance between the lenses 204, 206 may improve the overlap inthe fields-of-view 230, 234. In another embodiment (not shown), thelenses 204, 206 may be aligned along a common imaging axis.

Images or frames captured by the image capture devices 226, 228 may becombined, merged, or stitched together to produce a combined image, suchas a spherical or panoramic image, which may be an equirectangularplanar image. In some implementations, generating a combined image mayinclude use of techniques including noise reduction, tone mapping, whitebalancing, or other image correction. In some implementations, pixelsalong the stitch boundary may be matched accurately to minimize boundarydiscontinuities.

The image capture device 200 may be used to implement some or all of thetechniques described in this disclosure, such as the technique 700described in FIG. 7 or technique 800 described in FIG. 8 .

FIG. 3 is a block diagram of electronic components in an image capturedevice 300. The image capture device 300 may be a single-lens imagecapture device, a multi-lens image capture device, or variationsthereof, including an image capture device with multiple capabilitiessuch as use of interchangeable integrated sensor lens assemblies. Thedescription of the image capture device 300 is also applicable to theimage capture devices 100, 200 of FIGS. 1A-1B and 2A-2D.

The image capture device 300 includes a body 302 which includeselectronic components such as capture components 310, a processingapparatus 320, data interface components 330, movement sensors 340,power components 350, and/or user interface components 360.

The capture components 310 include one or more image sensors 312 forcapturing images and one or more microphones 314 for capturing audio.

The image sensor(s) 312 is configured to detect light of a certainspectrum (e.g., the visible spectrum or the infrared spectrum) andconvey information constituting an image as electrical signals (e.g.,analog or digital signals). The image sensor(s) 312 detects lightincident through a lens coupled or connected to the body 302. The imagesensor(s) 312 may be any suitable type of image sensor, such as acharge-coupled device (CCD) sensor, active pixel sensor (APS),complementary metal-oxide-semiconductor (CMOS) sensor, N-typemetal-oxide-semiconductor (NMOS) sensor, and/or any other image sensoror combination of image sensors. Image signals from the image sensor(s)312 may be passed to other electronic components of the image capturedevice 300 via a bus 380, such as to the processing apparatus 320. Insome implementations, the image sensor(s) 312 includes adigital-to-analog converter. A multi-lens variation of the image capturedevice 300 can include multiple image sensors 312.

The microphone(s) 314 is configured to detect sound, which may berecorded in conjunction with capturing images to form a video. Themicrophone(s) 314 may also detect sound in order to receive audiblecommands to control the image capture device 300.

The processing apparatus 320 may be configured to perform image signalprocessing (e.g., filtering, tone mapping, stitching, and/or encoding)to generate output images based on image data from the image sensor(s)312. The processing apparatus 320 may include one or more processorshaving single or multiple processing cores. In some implementations, theprocessing apparatus 320 may include an application specific integratedcircuit (ASIC). For example, the processing apparatus 320 may include acustom image signal processor. The processing apparatus 320 may includean attenuation map. The processing apparatus 320 may exchange data(e.g., image data) with other components of the image capture device300, such as the image sensor(s) 312, via the bus 380.

The processing apparatus 320 may include memory, such as a random-accessmemory (RAM) device, flash memory, or another suitable type of storagedevice, such as a non-transitory computer-readable memory. The memory ofthe processing apparatus 320 may include executable instructions anddata that can be accessed by one or more processors of the processingapparatus 320. For example, the processing apparatus 320 may include oneor more dynamic random-access memory (DRAM) modules, such as double datarate synchronous dynamic random-access memory (DDR SDRAM). In someimplementations, the processing apparatus 320 may include a digitalsignal processor (DSP). More than one processing apparatus may also bepresent or associated with the image capture device 300.

The data interface components 330 enable communication between the imagecapture device 300 and other electronic devices, such as a remotecontrol, a smartphone, a tablet computer, a laptop computer, a desktopcomputer, or a storage device. For example, the data interfacecomponents 330 may be used to receive commands to operate the imagecapture device 300, transfer image data to other electronic devices,and/or transfer other signals or information to and from the imagecapture device 300. The data interface components 330 may be configuredfor wired and/or wireless communication. For example, the data interfacecomponents 330 may include an I/O interface 332 that provides wiredcommunication for the image capture device, which may be a USB interface(e.g., USB type-C), a high-definition multimedia interface (HDMI), or aFireWire interface. The data interface components 330 may include awireless data interface 334 that provides wireless communication for theimage capture device 300, such as a Bluetooth interface, a ZigBeeinterface, and/or a Wi-Fi interface. The data interface components 330may include a storage interface 336, such as a memory card slotconfigured to receive and operatively couple to a storage device (e.g.,a memory card) for data transfer with the image capture device 300(e.g., for storing captured images and/or recorded audio and video).

The movement sensors 340 may detect the position and movement of theimage capture device 300. The movement sensors 340 may include aposition sensor 342, an accelerometer 344, or a gyroscope 346. Theposition sensor 342, such as a global positioning system (GPS) sensor,is used to determine a position of the image capture device 300. Theaccelerometer 344, such as a three-axis accelerometer, measures linearmotion (e.g., linear acceleration) of the image capture device 300. Thegyroscope 346, such as a three-axis gyroscope, measures rotationalmotion (e.g., rate of rotation) of the image capture device 300. Othertypes of movement sensors 340 may also be present or associated with theimage capture device 300.

The power components 350 may receive, store, and/or provide power foroperating the image capture device 300. The power components 350 mayinclude a battery interface 352 and a battery 354. The battery interface352 operatively couples to the battery 354, for example, with conductivecontacts to transfer power from the battery 354 to the other electroniccomponents of the image capture device 300. The power components 350 mayalso include the I/O interface 332, as indicated in dotted line, and thepower components 350 may receive power from an external source, such asa wall plug or external battery, for operating the image capture device300 and/or charging the battery 354 of the image capture device 300.

The user interface components 360 may allow the user to interact withthe image capture device 300, for example, providing outputs to the userand receiving inputs from the user. The user interface components 360may include visual output components 362 to visually communicateinformation and/or present captured images to the user. The visualoutput components 362 may include one or more lights 364 and/or moredisplays 366. The display(s) 366 may be configured as a touch screenthat receives inputs from the user. The user interface components 360may also include one or more speakers 368. The speaker(s) 368 canfunction as an audio output component that audibly communicatesinformation and/or presents recorded audio to the user. The userinterface components 360 may also include one or more physical inputinterfaces 370 that are physically manipulated by the user to provideinput to the image capture device 300. The physical input interfaces 370may, for example, be configured as buttons, toggles, or switches. Theuser interface components 360 may also be considered to include themicrophone(s) 314, as indicated in dotted line, and the microphone(s)314 may function to receive audio inputs from the user, such as voicecommands.

The image capture device 300 may be used to implement some or all of thetechniques described in this disclosure, such as the technique 700described in FIG. 7 or technique 800 described in FIG. 8 .

FIG. 4 is a diagram of an example of a tone mapping process 400 appliedin a pair of fish-eye images. As shown in FIG. 4 , the tone mappingprocess 400 includes obtaining a or one, e.g., a firsthyper-hemispherical image 410A from a first image sensor and another,e.g., a second hyper-hemispherical image 410B from a second imagesensor. Each hyper-hemispherical image 410A and 410B includes an imagecircle portion 420A and 420B, respectively, and a dark corner portion430A and 430B, respectively. The dark corner portions 430A and 430B donot contain any image data and are formed when hyper-hemisphericalimages are projected onto a rectangular image sensor.

Each image circle portion 420A and 420B includes a stitch line 440A and440B, respectively. The image circle portions 420A and 420B are stitchedtogether at the stitch lines to obtain a 360 image.

The tone mapping process 400 includes analyzing, for each pixel, aneighborhood of the pixel. Each neighborhood may be referred to as ablock, and may be of any dimension and contain any number of pixels. Inone example, each block may include a wide neighborhood of 100×100pixels. The pixel being analyzed may be at or near the center of theneighborhood or block. For each pixel, a weighted average luminance iscomputed using neighbor pixels in the block centered on that pixel. Thisprocessing may be referred to as sliding window processing or imagefiltering, where each analyzed pixel of the output image is the resultof the same function applied to all the pixels in the neighborhood ofthe analyzed pixel. The tone mapping process 400 may process the blocksthat contain a portion of an image circle, such as blocks A, B, and C.The tone mapping process 400 may ignore blocks that do not contain aportion of an image circle, such as block D. In this example, block Dwould not be processed.

As shown in FIG. 4 , the tone mapping process 400 includes performingLTM on a portion 450 of the image circle (shown in cross-hatching) wherethe block does not overlap with the stitch line 440A, for example, blockC. LTM techniques may change the pixel intensity depending on theintensity of its neighboring pixels per the following:

g _(LTM)(x)=Φ({Y(t) s.t. t∈N(x)})  Equation (1)

where N(x) is the set of pixels in the neighborhood of x, Y (t) is theluminance of pixel t and Φ is the function for computation ofg_(LTM)(x). LTM may be used to preserve or enhance details in the image,but it can also change the global exposure. For example, LTM may be usedto remove low frequency variations, preserve or enhance high frequencydetails, or both. In the example LTM algorithm, the gain depends ony(x), a filtered version of the image luminance component. This filteredluminance also depends on the neighborhood of pixel x.

The tone mapping process 400 includes performing GTM on a portion 460 ofthe image circle (shown in stippling) where the block, such as block A,overlaps with the stitch line 440A. GTM techniques may include theapplication of a gain g_(GTM)(x) at pixel x determined by the ratio

$\begin{matrix}{{g_{GTM}(x)} = \frac{{TC}\left( {Y(x)} \right)}{Y(x)}} & {{Equation}(2)}\end{matrix}$

where the techniques may depend on a tone curve TC and luminance Y(x) ofpixel x.

As shown in FIG. 4 , the LTM processing is performed in the areas thatare not impacted by border or continuity issues, such as areas near thecenter of the image circles 420A and 420B (e.g., block C) and GTM isperformed in the areas that are impacted by border or continuity issues,such as portion 460. GTM may be performed when the neighborhood or blockintersects with the stitch line 440A or 440B or the dark corners 430A or430B of the image. In some examples, a combination of LTM and GTM may beperformed in a portion 470 of the image circle between portion 450 andportion 460. In portion 470, the amount of LTM and GTM applied may bebased on an attenuation map. The attenuation map may be based on thedistance of a pixel from the center of the image circle. For example,the amount of GTM applied may progressively increase according to theattenuation map as the pixel being processed approaches the stitch line440A.

FIG. 5 is a graph of an example of a radial profile of an attenuationmap 500. The attenuation map shows an attenuation coefficient functionbased on a normalized distance to an image center. As shown in FIG. 5 ,the x-axis indicates the normalized distance to the image center, where0 represents the center of the image circle and 1 represents the outeredge of the image circle. The y-axis indicates the attenuationcoefficient, where 0 represents no attenuation (i.e., 100% LTMapplication) and 1 represents 100% attenuation (i.e., 100% GTMapplication). It can be seen that no attenuation is applied near thecenter of the image circle, for example the pixels at point 510. As thedistance of the pixels increase from the center of the image circle, GTMis progressively applied, for example the pixels at point 520. As thestitch line is approached, 100% GTM is applied to the pixels at point530.

FIG. 6 is a diagram of an example of a tone mapping process 600 appliedin an EAC format image. As shown in FIG. 6 , the tone mapping process600 includes obtaining a first hyper-hemispherical image from a firstimage sensor, shown as a first band 610A, and a secondhyper-hemispherical image from a second image sensor, shown as a secondband 610B.

Each band 610A and 610B includes a stitch line 620A and 620B,respectively. As shown in FIG. 6 , the stitch lines 620A and 620Bcorrespond to the outer perimeter of their respective bands. The bands610A and 610B are stitched together at the stitch lines to obtain a 360image represented by six faces of a cube.

The tone mapping process 600 includes analyzing, for each pixel, aneighborhood of the pixel. Each neighborhood may be referred to as ablock, and may be of any dimension and contain any number of pixels. Inone example, each block may include a wide neighborhood of 100×100pixels. The pixel being analyzed may be at or near the center of theneighborhood or block. For each pixel, a weighted average luminance iscomputed using neighbor pixels in the block centered on that pixel. Thisprocessing may be referred to as sliding window processing or imagefiltering, where each analyzed pixel of the output image is the resultof the same function applied to all the pixels in the neighborhood ofthe analyzed pixel.

As shown in FIG. 6 , the tone mapping process 600 includes performingLTM on a portion 630 of each band (shown in cross-hatching) where therespective blocks do not overlap with the respective stitch lines 620Aand 620B. The LTM techniques applied may change the pixel intensitydepending on the intensity of its neighboring pixels per Equation (1)above. LTM may be used to preserve or enhance details in the image, butit can also change the global exposure. In the example LTM algorithm,the gain depends on a filtered version of the image luminance component.This filtered luminance also depends on the neighborhood of pixel x.

The tone mapping process 600 includes performing GTM on a portion 640 ofeach band (shown in stippling) where a respective block, such as blockA, overlaps with the respective stitch lines 620A or 620B. GTM may beperformed on a block, such as block B, that intersects with the otherband or the boundary of EAC image. GTM techniques may include theapplication of a gain g_(GTM)(x) at pixel x determined by the ratioshown in Equation (2) above.

As shown in FIG. 6 , the LTM processing is performed in the areas thatare not impacted by border or continuity issues, such as areas near thecenter of the bands 610A and 610B and GTM is performed in the areas thatare impacted by border or continuity issues, such as portion 640. Insome examples, a combination of LTM and GTM may be performed in aportion 650 of the respective bands between portion 630 and portion 640.In portion 650, the amount of LTM and GTM applied may be based on anattenuation map. The attenuation map may be based on the distance of apixel from the outer perimeter of each respective band. For example, theamount of GTM applied may progressively increase according to theattenuation map as the pixel being processed approaches the stitch line620A and 620B.

FIG. 7 is a flow diagram of an example of a method 700 for fieldvariable tone mapping. As shown in FIG. 7 , the method 700 includesobtaining 710 one or more hyper-hemispherical images, for example, afirst hyper-hemispherical image from a first image sensor and a secondhyper-hemispherical image from a second image sensor. Thehyper-hemispherical images may be in a fish-eye format, and ERP imageformat, or an EAC image format. In a fish-eye format example, eachhyper-hemispherical image includes an image circle portion and a darkcorner portion. The dark corner portions do not contain any image dataand are formed when hyper-hemispherical images are projected onto arectangular sensor.

Each hyper-hemispherical image includes a stitch line. In a fish-eyeformat example, each image circle portion contains a stitch line at ornear the outer circumference of the image circle portion. A radius ofthe image circle portion may be larger than a radius of the stitch line.In an EAC image format example, the outer perimeter of each band is thestitch line. The hyper-hemispherical images are stitched together at thestitch lines to obtain a 360 image.

The method 700 includes performing 720 LTM on a first area of thehyper-hemispherical image. The first area of the hyper-hemisphericalimage may be an area where a block of a predetermined size does notoverlap with the stitch line. The block may correspond to aneighborhood, and may include, for example, a pixel area ofapproximately 100×100 pixels. LTM techniques may change the pixelintensity depending on the intensity of its neighboring pixels perEquation (1) above. LTM may be used to preserve or enhance details inthe image, but it can also change the global exposure. In the exampleLTM algorithm, the gain depends on a filtered version of the imageluminance component. This filtered luminance also depends on theneighborhood of the pixel.

The method 700 includes performing 730 GTM on a second area of thehyper-hemispherical image where the block overlaps with the stitch line.GTM techniques may include the application of a gain g_(GTM)(x) at pixelx determined by the ratio shown in Equation (2) above.

The LTM processing is performed in the areas that are not impacted byborder or continuity issues, such as areas near the center of thehyper-hemispherical images and GTM is performed in the areas that areimpacted by border or continuity issues. In some examples, a combinationof LTM and GTM may be performed in a portion of the hyper-hemisphericalimages. In these examples, the amount of LTM and GTM applied may bebased on an attenuation map. The attenuation map may be based on thedistance of a pixel from the center of the image circle, or the distanceof a pixel from the stitch line. For example, the amount of GTM appliedmay progressively increase according to the attenuation map as the pixelbeing processed approaches the stitch line.

The method 700 includes stitching 740 the hyper-hemispherical images toobtain a processed image. The processed image may include 360 content.The processed image may be stored or output 750 in any 360 output imageformat, for example, a paired fish-eye image format, an ERP imageformat, or in an EAC image format.

FIG. 8 is a flow diagram of an example of another method 800 for fieldvariable tone mapping. The method 800 includes obtaining 810 one or morehyper-hemispherical images, for example, a first hyper-hemisphericalimage from a first image sensor and a second hyper-hemispherical imagefrom a second image sensor. The hyper-hemispherical images may be in afish-eye format, and ERP image format, or an EAC image format. In afish-eye format example, each hyper-hemispherical image includes animage circle portion and a dark corner portion. The dark corner portionsdo not contain any image data and are formed when hyper-hemisphericalimages are projected onto a rectangular sensor.

Each hyper-hemispherical image includes a stitch line. In a fish-eyeformat example, each image circle portion contains a stitch line at ornear the outer circumference of the image circle portion. A radius ofthe image circle portion may be larger than a radius of the stitch line.In an EAC image format example, the outer perimeter of each band is thestitch line. The hyper-hemispherical images are stitched together at thestitch lines to obtain a 360 image.

The method 800 includes analyzing 820, for each pixel, a neighborhood ofthe pixel. Each neighborhood may be referred to as a block, and may beof any dimension and contain any number of pixels. In one example, eachblock may include a wide neighborhood of approximately 100×100 pixels.The pixel being analyzed may be at or near the center of theneighborhood or block.

In a fish-eye format example, method 800 includes determining 830whether a neighborhood or block contains a portion of an image circle.If the block does not contain a portion of the image circle, the method800 may include ignoring 840 the block and continuing to the next block.In an EAC image format example, the steps 830 and 840 may be skippedsince this format does not include an image circle.

The method 800 includes determining 850 whether the block overlaps withthe stitch line, an image border of the EAC image, an image border ofanother band, or any combination thereof. If the block does not overlapwith the stitch line, the image border, or the other band, the method800 includes computing a neighborhood luminance for a pixel. The pixelmay be a center pixel in a neighborhood of pixels. A neighborhood may beof any dimension and contain any number of pixels. In one example, theneighborhood may be a wide neighborhood of approximately 100×100 pixels.The neighborhood luminance may be a weighted average luminance of apredetermined number of pixels, for example, the neighborhood of pixels.A non-linear filter, such as a bilateral filter or a guided filter, maybe used to preserve the large luminance transitions and obtain theweighted average luminance. As shown in FIG. 8 , the method includesapplying 865 a gain to the pixel. The gain applied to the pixel may bebased on the neighborhood luminance. The method 800 may then continue tothe next block.

If the block does overlap with the stitch line, the image border, or theother band, the method 800 includes determining 870 a distance of thepixel from the center of the hyper-hemispherical image. The method 800includes computing 875 a pixel luminance and applying 880 a gain to thepixel based on the pixel luminance. In an example, the method 800 mayinclude applying 890 an attenuation map. The attenuation map may bebased on the determined distance of the pixel from the center of thehyper-hemispherical image, a determined distance of the pixel from thestitch line, or both. The method 800 may then continue to the nextblock.

The attenuation map may be applied to smoothed luminance, local contrastenhancement strength, or both. Let M_(att)(x) be the value of the radialattenuation map at pixel x in interval [0, 1]. The smoothed luminance isblended with the pixel luminance by:

y _(ref)(x)=(1−M _(att)(x))× y (x)+M _(att)(x)×y(x)  Equation (3)

in order to use y_(ref)(x) in the gain calculation block for tonemapping of the low frequency part of the image, as shown below inEquation (4).

$\begin{matrix}{{g_{{LTM},1}(x)} = \frac{{TC}\left( {y_{ref}(x)} \right)}{y_{ref}(x)}} & {{Equation}(4)}\end{matrix}$

The attenuation map may also be used for the reduction of detailsenhancement by reducing the related coefficients G_(mid) and G_(high)respectively for mid-frequencies and high-frequencies enhancement, asshown below in Equation (5) and Equation (6).

G′ _(mid)(x)=(1−M _(att)(x))×G _(mid)(x)+M _(att)(x)  Equation (5)

G′ _(high)(x)=(1−M _(att)(x))×G _(high)(x)+M _(att)(x)  Equation (6)

When the attenuation map is equal to 1, the detail enhancement gains arealso equal to 1. A second gain is computed for details enhancementg_(LTM,2)(x) as a function of y(x), y(x), G′_(mid)(x) and G′_(high)(x).The LTM gain may be the product of g_(LTM,1)(x) and g_(LTM,2)(x), asshown below in Equation (7).

g _(LTM)(x)=g _(LTM,1)(x)×g _(LTM,2)(x)  Equation (7)

In one or more embodiments described herein, a spatial attenuation mapmay be used to reduce the effect of LTM inside regions of interest. Aregion of interest may include an area of the image that includes aparticular object, such as a face, trees, sky, or the like. For example,the LTM block may use face detection information to reduce LTM effectson faces to avoid unpleasant images. In one or more embodiments, aradial spatial attenuation map may be used to reduce the effect of LTMinside regions of interest in pairs of fish-eye images. Both radial andspatial attenuation maps may be implemented in hardware, software, orboth, and may be applied on standard rectangular images, circularimages, or both.

While the disclosure has been described in connection with certainembodiments, it is to be understood that the disclosure is not to belimited to the disclosed embodiments but, on the contrary, is intendedto cover various modifications and equivalent arrangements includedwithin the scope of the appended claims, which scope is to be accordedthe broadest interpretation so as to encompass all such modificationsand equivalent structures as is permitted under the law.

What is claimed is:
 1. An image capture device comprising: an imagesensor configured to obtain a hyper-hemispherical image; and a processorconfigured to: perform local tone mapping (LTM) on a first area ofpixels of the hyper-hemispherical image; perform global tone mapping(GTM) on a second area of pixels of the hyper-hemispherical image toobtain a processed image, wherein an amount of LTM performedprogressively converges to an amount of GTM performed in a third area ofpixels that is between the first area of pixels and the second area ofpixels; and output the processed image.
 2. The image capture device ofclaim 1, wherein the processor is further configured to perform LTM oneach pixel of the first area of pixels based on a predefined area ofpixels.
 3. The image capture device of claim 2, wherein the predefinedarea of pixels is a 100×100 pixel area.
 4. The image capture device ofclaim 1, wherein the processor is further configured to perform LTM oneach pixel of the first area of pixels to remove low frequencyvariations.
 5. The image capture device of claim 4, wherein theprocessor is further configured to perform LTM on each pixel of thefirst area of pixels to preserve high frequency details.
 6. The imagecapture device of claim 1, further comprising: a second image sensorconfigured to obtain a second hyper-hemispherical image; wherein theprocessor is further configured to output the processed image in a 360output format.
 7. The image capture device of claim 6, wherein the 360output format is a stitched pair fish-eye format, an equi-angularcubemap (EAC) format, or an equi-rectangular projection (ERP) format. 8.A method comprising: obtaining a hyper-hemispherical image that has afirst image portion; dividing the hyper-hemispherical image into aplurality of blocks; determining whether a block of the plurality ofblocks contains a portion of the first image portion; determiningwhether the block overlaps a second image portion; computing aneighborhood luminance of a pixel and applying a gain to the pixel basedon the neighborhood luminance when the block does not overlap with thesecond image portion; when the block overlaps with the second imageportion, the method further comprising: determining a distance of thepixel from a center of the first image portion; computing a luminance ofthe pixel; applying a gain to the pixel based on the luminance; applyingan attenuation map to the pixel based on the determined distance of thepixel from the center of the first image portion to obtain a processedimage; and outputting the processed image.
 9. The method of claim 8further comprising: dividing the other hyper-hemispherical image into aplurality of blocks, wherein each of the plurality of blocks contains apredetermined number of pixels.
 10. The method of claim 8, wherein theneighborhood luminance is a weighted average luminance of apredetermined number of pixels.
 11. The method of claim 8, wherein eachblock of the plurality of blocks has is a 100×100 pixel area.
 12. Themethod of claim 8, wherein the attenuation map is applied to a smoothedluminance of the pixel.
 13. The method of claim 8, wherein theattenuation map is applied to a local contrast enhancement strength ofthe pixel.
 14. An image capture device comprising: a first image sensorconfigured to obtain a first hyper-hemispherical image that has a firstimage portion; a second image sensor configured to obtain a secondhyper-hemispherical image that has a second image portion; and aprocessor configured to: perform local tone mapping (LTM) on a firstarea of pixels of the first image portion; perform global tone mapping(GTM) on a second area of pixels of the first image portion to obtain aprocessed image, wherein an amount of LTM performed converges to anamount of GTM performed; and output the processed image.
 15. The imagecapture device of claim 14, wherein the processor is further configuredto: perform LTM on a third area of pixels of the second image portion;and perform GTM on a fourth area of pixels of the second image portionwhen a portion of the predefined area of pixels overlaps with the firstimage portion.
 16. The image capture device of claim 15, wherein theprocessor is further configured to perform LTM on each pixel of thefirst area of pixels and the third area of pixels based on thepredefined area of pixels.
 17. The image capture device of claim 15,wherein the predefined area of pixels is a 100×100 pixel area.
 18. Theimage capture device of claim 15, wherein the processor is furtherconfigured to perform LTM on each pixel of the first area of pixels andthe third area of pixels to remove low frequency variations.
 19. Theimage capture device of claim 18, wherein the processor is furtherconfigured to perform LTM on each pixel of the first area of pixels andthe third area of pixels to preserve high frequency details.
 20. Theimage capture device of claim 14, wherein the processor is furtherconfigured to: perform a combination of LTM and GTM on a fifth area ofpixels of the first image portion, wherein the fifth area of pixels isbetween the first area of pixels and the second area of pixels, andwherein the combination is based on an attenuation map.